TRANSOM: An Efficient Fault-Tolerant System for Training LLMs

Large language models (LLMs) with hundreds of billions or trillions of parameters, represented by chatGPT, have achieved profound impact on various fields. However, training LLMs with super-large-scale parameters requires large high-performance GPU clusters and long training periods lasting for months. Due to the inevitable hardware and software failures in large-scale clusters, maintaining uninterrupted and long-duration training is extremely challenging. As a result, A substantial amount of training time is devoted to task checkpoint saving and loading, task rescheduling and restart, and task manual anomaly checks, which greatly harms the overall training efficiency. To address these issues, we propose TRANSOM, a novel fault-tolerant LLM training system. In this work, we design three key subsystems: the training pipeline automatic fault tolerance and recovery mechanism named Transom Operator and Launcher (TOL), the training task multi-dimensional metric automatic anomaly detection system named Transom Eagle Eye (TEE), and the training checkpoint asynchronous access automatic fault tolerance and recovery technology named Transom Checkpoint Engine (TCE). Here, TOL manages the lifecycle of training tasks, while TEE is responsible for task monitoring and anomaly reporting. TEE detects training anomalies and reports them to TOL, who automatically enters the fault tolerance strategy to eliminate abnormal nodes and restart the training task. And the asynchronous checkpoint saving and loading functionality provided by TCE greatly shorten the fault tolerance overhead. The experimental results indicate that TRANSOM significantly enhances the efficiency of large-scale LLM training on clusters. Specifically, the pre-training time for GPT3-175B has been reduced by 28%, while checkpoint saving and loading performance have improved by a factor of 20.Comment: 14 pages, 9 figure

Theoretical foundations of studying criticality in the brain

Criticality is hypothesized as a physical mechanism underlying efficient transitions between cortical states and remarkable information processing capacities in the brain. While considerable evidence generally supports this hypothesis, non-negligible controversies persist regarding the ubiquity of criticality in neural dynamics and its role in information processing. Validity issues frequently arise during identifying potential brain criticality from empirical data. Moreover, the functional benefits implied by brain criticality are frequently misconceived or unduly generalized. These problems stem from the non-triviality and immaturity of the physical theories that analytically derive brain criticality and the statistic techniques that estimate brain criticality from empirical data. To help solve these problems, we present a systematic review and reformulate the foundations of studying brain criticality, i.e., ordinary criticality (OC), quasi-criticality (qC), self-organized criticality (SOC), and self-organized quasi-criticality (SOqC), using the terminology of neuroscience. We offer accessible explanations of the physical theories and statistic techniques of brain criticality, providing step-by-step derivations to characterize neural dynamics as a physical system with avalanches. We summarize error-prone details and existing limitations in brain criticality analysis and suggest possible solutions. Moreover, we present a forward-looking perspective on how optimizing the foundations of studying brain criticality can deepen our understanding of various neuroscience questions

Evolutionary many-objective optimization：A survey

Many-objective optimization problems (MaOPs) widely exist in industrial and scientific fields, where there are more than 3 objectives that are conflicting with each other (i.e., the improvement of the performance in one objective may lead to the deterioration of the performance of some other objectives). Because of the conflict between objectives, there is no unique optimal solution for MaOPs, but a group of compromise solutions need to be obtained to balance between objectives. As a class of population-based optimization algorithms inspired by biological evolution principles evolutionary algorithms have been proved to be effective in solving MaOPs, and have become one of the research hot spots in the field of multi-objective optimization. In the past 20 years, the research on many-objective evolutionary algorithms (MaOEAs) has made great progress, and a large number of advanced evolutionary methods and evaluation systems have been proposed and improved. In this paper, the research progress of evolutionary many-objective optimization (EMaO) is comprehensively reviewed. Specifically, it includes: (1) Describing the relevant theoretical background of EMaO; (2) Analyzing the problems and challenges faced by evolutionary algorithms in solving MaOPs; (3) Discussing the development of MaOEAs in detail; (4) Summarizing MaOPs and performance indicators in detail; (5) Introducing the visualization tools for high-dimensional objective space; (6) Summarizing the application of MaOEAs in some fields, and (7) Providing suggestions for future research in the domain

Time to Clinical Benefit of Intensive Blood Pressure Lowering in Patients 60 Years and Older With Hypertension

Importance Recent guidelines recommend a systolic blood pressure (BP) goal of less than 150 mm Hg or even 130 mm Hg for adults aged 60 years or older. However, harms from intensive BP treatments occur immediately (eg, syncope, fall), and benefits for cardiovascular event reduction emerge over time. Therefore, harms with low chance of benefit need to be clearer, particularly for those with limited life expectancy. Objective To estimate the time needed to potentially derive clinical benefit from intensive BP treatment in patients 60 years and older. Design, Setting, and Participants This secondary analysis included individual patient data from published randomized clinical trials with 27 414 patients 60 years or older with hypertension. Patient-level survival data were reconstructed when the original data were not available. Published trials were identified by searching PubMed until October 15, 2021. Exposures Intensive BP lowering vs standard BP lowering with the treat-to-target design. Main Outcomes and Measures Major adverse cardiovascular event (MACE) defined by each trial, which was broadly similar with all trials including myocardial infarction, stroke, and cardiovascular mortality. Results Six trials (original data from 2 trials and reconstructed data from 4 trials) with 27 414 participants (mean age, 70 years; 56.3% were women) were included in the analysis. Intensive BP treatment with a systolic BP target below 140 mm Hg was significantly associated with a 21% reduction in MACE (hazard ratio, 0.79; 95% CI, 0.71-0.88; P < .001). On average, 9.1 (95% CI, 4.0-20.6) months were needed to prevent 1 MACE per 500 patients with the intensive BP treatment (absolute risk reduction [ARR], 0.002). Likewise, 19.1 (95% CI, 10.9-34.2) and 34.4 (95% CI, 22.7-59.8) months were estimated to avoid 1 MACE per 200 (ARR, 0.005) and 100 (ARR, 0.01) patients, respectively. Conclusions and Relevance In this analysis, findings suggest that for patients 60 years and older with hypertension, intensive BP treatment may be appropriate for some adults with a life expectancy of greater than 3 years but may not be suitable for those with less than 1 year

Plasma exosomal miR-320d, miR-4479, and miR-6763-5p as diagnostic biomarkers in epithelial ovarian cancer

BackgroundExosomal miRNA had been proved as the promising biomarkers for multiple cancers including epithelial ovarian cancer (EOC). This study aimed to validate the diagnostic accuracy of exosomal miR-320d, miR-4479, and miR-6763-5p for EOC.Materials and methodsExosomes isolated from the plasma by ultracentrifugation were verified using TEM, qNano and western blot. MiRNAs sequencing was used to screen out the differential exosomal miRNAs and miR-320d, miR-4479, and miR-6763-5p were selected as candidates, which were further verified by RT-qPCR in 168 healthy donors and 161 primary EOC patients. Besides, the diagnostic accuracy of these three exosomal miRNAs were evaluated using the receiver operating characteristic curve (ROC).ResultsMiRNAs sequencing revealed 95 differential exosomal miRNAs between EOC patients and healthy donors. Subsequently, exosomal miR-320d, miR-4479, and miR-6763-5p were significantly down regulated in EOC patients compared with healthy controls and benign patients. More importantly, these three miRNAs could serve as circulating diagnostics biomarkers for EOC, possessing areas under the curve (AUC) of 0.6549, 0.7781, and 0.6834, respectively. Moreover, these three exosomal miRNAs levels were closely associated with lymph node metastasis, meanwhile exosomal miR-320d and miR-4479 expression was related to tumor stage.ConclusionExosomal miR-320d, miR-4479, and miR-6763-5p might serve as potential biomarkers for EOC

Time‐weighted blood pressure with cardiovascular risk among patients with or without diabetes

Background: Usual measures of blood pressure (BP) do not account for both the magnitude and duration of exposure to elevated BP over time. We aimed to demonstrate the effect of a novel time‐weighted BP on cardiovascular outcomes using a post hoc analysis of two published randomized trials. Hypothesis: Time‐weighted blood pressure is associated with cardiovascular risk among patients with or without diabetes. Methods: The limited‐access ACCORD and SPRINT data sets were used for the current study. Time‐weighted BP is obtained by dividing cumulative BP by the total follow‐up time. Time‐weighted BP burden above a threshold is also determined after deriving the time‐weighted BP by re‐zeroing the interpolated pressure values at two different hypertension thresholds (>140/90 and >130/80 mmHg). Results: Eighteen thousand five hundred forty‐one patients from the two clinical trials were enrolled in this study. A J‐curve relation was observed between time‐weighted BP and major cardiovascular events (MACE). The systolic blood pressure (SBP) burden independently predicted MACE across the two trials at different thresholds (ACCORD: SBP > 130 mmHg, HR = 1.05 [1.03−1.06]; SBP > 140 mmHg, HR = 1.06 [1.04−1.08]; SPRINT: SBP > 130 mmHg, HR = 1.04 [1.03−1.05]; SBP > 140 mmHg, HR = 1.05 [1.04−1.07]). Consistent results were found for diastolic blood pressure (DBP) burden (ACCORD: DBP > 80 mmHg, HR = 1.10 [1.06−1.15]; DBP > 90 mmHg, HR = 1.20 [1.11−1.30]. SPRINT: DBP > 80 mmHg, HR = 1.06 [1.02−1.09]; DBP > 90 mmHg, HR = 1.12 [1.06−1.18]). Significant associations were also observed for stroke, myocardial infarction, cardiovascular death, and all‐cause mortality. Conclusion: Both time‐weighted SBP and DBP independently influenced the risk of adverse cardiovascular events among patients with and without diabetes, regardless of the definition of hypertension (130/80 or <140/90 mmHg)

Discovering cancer genes by integrating network and functional properties

<p>Abstract</p> <p>Background</p> <p>Identification of novel cancer-causing genes is one of the main goals in cancer research. The rapid accumulation of genome-wide protein-protein interaction (PPI) data in humans has provided a new basis for studying the topological features of cancer genes in cellular networks. It is important to integrate multiple genomic data sources, including PPI networks, protein domains and Gene Ontology (GO) annotations, to facilitate the identification of cancer genes.</p> <p>Methods</p> <p>Topological features of the PPI network, as well as protein domain compositions, enrichment of gene ontology categories, sequence and evolutionary conservation features were extracted and compared between cancer genes and other genes. The predictive power of various classifiers for identification of cancer genes was evaluated by cross validation. Experimental validation of a subset of the prediction results was conducted using siRNA knockdown and viability assays in human colon cancer cell line DLD-1.</p> <p>Results</p> <p>Cross validation demonstrated advantageous performance of classifiers based on support vector machines (SVMs) with the inclusion of the topological features from the PPI network, protein domain compositions and GO annotations. We then applied the trained SVM classifier to human genes to prioritize putative cancer genes. siRNA knock-down of several SVM predicted cancer genes displayed greatly reduced cell viability in human colon cancer cell line DLD-1.</p> <p>Conclusion</p> <p>Topological features of PPI networks, protein domain compositions and GO annotations are good predictors of cancer genes. The SVM classifier integrates multiple features and as such is useful for prioritizing candidate cancer genes for experimental validations.</p

Evolution Of Drosophila Ribosomal Protein Gene Core Promoters

The coordinated expression of ribosomal protein genes (RPGs) has been well documented in many species. Previous analyses of RPG promoters focus only on Fungi and mammals. Recognizing this gap and using a comparative genomics approach, we utilize a motif-finding algorithm that incorporates cross-species conservation to identify several significant motifs in Drosophila RPG promoters. As a result, significant differences of the enriched motifs in RPG promoter are found among Drosophila, Fungi, and mammals, demonstrating the evolutionary dynamics of the ribosomal gene regulatory network. We also report a motif present in similar numbers of RPGs among Drosophila species which does not appear to be conserved at the individual RPG gene level. A module-wise stabilizing selection theory is proposed to explain this observation. Overall, our results provide significant insight into the fast-evolving nature of transcriptional regulation in the RPG module. © 2008 Elsevier B.V. All rights reserved

CoCALC: A Self-Supervised Visual Place Recognition Approach Combining Appearance and Geometric Information

Visual place recognition (VPR) is considered among the most complicated tasks in SLAM due to the multiple challenges of drastic variations in both appearance and viewpoint. To address this issue, this article presents a self-supervised and lightweight VPR approach (namely CoCALC) that fully utilizes the appearance and geometric information provided by images. The main thing that makes CoCALC ultra-lightweight (only 0.27 MB) is our use of Depthwise Separable Convolution (DSC), a simple but effective architecture that enables our model to generate a more robust image representation. The network trained specifically for VPR can efficiently extract deep convolutional features from salient image regions that have relatively higher entropy, thereby expanding its applications on resource-limited platforms without GPUs. To further eliminate the negative consequences of the high percent false matches, a novel band-matrix-based geometric check is employed to filter out the incorrect matching of image patches, and the impact of different bandwidths on the recall rate is discussed. Results on several benchmark datasets confirm that the proposed CoCALC can yield state-of-the-art performance and superior generalization with acceptable efficiency. All relevant codes are provided at https://github.com/LiKangyuLKY/CoCALC-VPR for further studies